Bayes-TrEx: a Bayesian Sampling Approach to Model Transparency by Example

Post-hoc explanation methods are gaining popularity for interpreting, understanding, and debugging neural networks. Most analyses using such explain decisions in response to inputs drawn from the test set. However, set may have few examples that trigger some model behaviors, as high-confidence failures or ambiguous classifications. To address these challenges, we introduce a flexible inspection framework: Bayes-TrEx. Given data distribution, Bayes-TrEx finds in-distribution which specified prediction confidence. We demonstrate several use cases of Bayes-TrEx, including revealing highly confident (mis)classifications, visualizing class boundaries via examples, understanding novel-class extrapolation behavior, exposing network overconfidence. study classifiers trained on CLEVR, MNIST, Fashion-MNIST, show this framework enables more holistic analysis than just inspecting Code supplemental material available at https://github.com/serenabooth/Bayes-TrEx.